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(S) Method and system for accessing associated data sets in a multimedia environment in a data 
processing system. 



(57) A method and system in a data processing 
system for selectively associating stored data 
with an animated element within a multimedia 
presentation in a data processing system. At 
least one data set is created or identified within 
the data processing system. The data set is then 
associated within the data processing system 
with a selected animated element within the 
multimedia presentation. Thereafter, during the 
multimedia presentation, the existence of the 
data set and the association between the data 
set and the selected animated element is 
graphically indicated utilizing a rectangle sur- 
rounding the animated element, a blinking fig- 
ure near the animated element or any other 
suitable graphic indicator. A user may elect to 
have the associated data set presented by 
selecting the additional graphic element as- 
sociated with the specified animated element, 
or by selecting a button or other graphic indi- 
cator, during a specified period of time follow- 
ing the graphic indication. 




LU 



Jouve, 18, rue Saint-Denis, 75001 PARIS 



1 



EP 0 596 823 A2 



2 



BACKGROUND OF THE INVENTION 

1. Technical Field: 

The present invention relates in general to an im- 5 
proved data processing system, and in particular to a 
method and system for associating data sets with a 
multimedia presentation operating within a data proc- 
essing system. Still more particularly, the present in- 
vention relates to a method and system for associat- w 
ing stored data sets with an animated element depict- 
ed in a multimedia presentation and for notifying a 
user of the existence of an associated data set during 
a relevant portion of the multimedia presentation. 

15 

2. Description of the Related Art: 

Computer systems are capable of communicat- 
ing information with humans in many formats, includ- 
ing: text, graphics, sounds (including high-fidelity 20 
stereo), animated graphics, synthesized speech, and 
video. A multimedia presentation utilizes a data proc- 
essing system to combine such information formats 
into often coherent and comprehensive presentation. 

In known data processing systems, an additional 25 
data set may be linked to textual information. For ex- 
ample, utilizing hypertext, the existence of an addi- 
tional data set associated with a selected portion of 
text may be indicated in several ways, such as high- 
lighting or underlining the selected text. The informa- 30 
tion associated with such selected text may also be 
accessed in several ways, for example, by utilizing a 
pointing device, such as a mouse, and clicking the 
mouse while the indicator is positioned on the high- 
lighted text. After selecting highlighted text, the user 35 
may be presented with a list of additional relevant tex- 
tual data sets. After choosing from such a list, addi- 
tional textual information may be displayed in a new 
window. 

For example, if the word elephant appears in an 40 
encyclopedic article about India (note text is under- 
lined or differentiated in some manner from the sur- 
rounding text), a user may elect to access an article 
about elephants by positioning a graphical indicator, 
such as a mouse pointer, over the word, and clicking 45 
the mouse button. This selection process may then 
open another window within the display containing 
textual information about elephants. 

In prior art systems, textual information having 
such a link to other data sets is static in nature. That 50 
is, the link with a particular passage of text always ex- 
ists. While hypertext works well for textual informa- 
tion, in a multimedia environment it would be benefi- 
cial to link a multimedia presentation to another rele- 
vant data set and to allow such a link to be accessed 55 
only during a relevant period of time. That is, the link 
with an associated data set may only be active during 
a relevant portion of the multimedia presentation. 



In view of the above, it should be appreciated that 
a need exists for a method and system for graphically 
indicating, during a relevant period of a multimedia 
presentation, the existence of a data set, stored in a 
data processing system, which is associated with an 
animated element within a multimedia presentation, 
wherein a system user may elect to view or listen to 
the associated data set by selecting one or more of 
the available data sets. 

SUMMARY OF THE INVENTION 

It is therefore one object of the present invention 
to provide an improved data processing system. 

It is another object of the present invention to pro- 
vide a method and system for associating data sets 
with a multimedia presentation operating within a 
data processing system. 

It is yet another object of the present invention to 
provide a method and system for associating stored 
data sets with an animated element depicted in a mul- 
timedia presentation and notifying a user of the exis- 
tence of an associated data set during a relevant por- 
tion of the multimedia presentation. 

The foregoing objects are achieved as is now de- 
scribed. A method and system are disclosed in a data 
processing system for selectively associating stored 
data with an animated element within a multimedia 
presentation in a data processing system. At least one 
data set is created or identified within the data proc- 
essing system. The data set is then associated within 
the data processing system with a selected animated 
element within the multimedia presentation. There- 
after, during the multimedia presentation, the exis- 
tence of the data set and the association between the 
data set and the selected animated element is graph- 
ically indicated utilizing a rectangle surrounding the 
animated element, a blinking figure near the animat- 
ed element or any other suitable graphic indicator. A 
user may elect to have the associated data set pre- 
sented by selecting t he additional graphic element as- 
sociated with the specified animated element, or by 
selecting a button or other graphic indicator, during a 
specified period of time following the graphic indica- 
tion. 

The above as well as additional objects, features, 
and advantages of the present invention will become 
apparent in the following detailed written description. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The novel features believed characteristic of the 
invention are setforth in the appended claims. The in- 
vention itself however, as well as a preferred mode of 
use, further objects and advantages thereof, will best 
be understood by reference to the following detailed 
description of an illustrativeembodimentwhen read in 
conjunction with the accompanying drawings, where- 
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in: 

Figure 1 depicts a pictorial illustration of a data 
processing system which may be utilized to im- 
plement the method and system of the present 
invention; 

Figure 2 is a high level block diagram further il- 
lustrating details of the data processing system 
as shown in Figure 1; 

Figure 3 depicts a pictorial representation of a 
data processing system display at three sequen- 
tial points in time during a multimedia presenta- 
tion; 

Figure 4 is a high level logical flow chart which 
depicts a programmed response of the data proc- 
essing system of Figure 1 in accordance with a 
preferred embodiment of the method and system 
of the present invention during a set-up opera- 
tion; 

Figure 5 is a high level logical flowchart which il- 
lustrates a programmed response of the data 
processing system of Figure 1 in accordance with 
a preferred embodiment of the method and sys- 
tem of the present invention during a graphic in- 
dication process; and 

Figure 6 is a high level logical flow chart which 
depicts a programmed response of the data proc- 
essing system of Figure 1 in accordance with a 
preferred embodiment of the method and system 
of the present invention during a user selection 
process. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENT 

With reference now to the figures and in particu- 
lar with reference to Figure 1, there is depicted a pic- 
torial illustration of a data processing system 50 which 
may be utilized to implement a preferred embodiment 
of the present invention. Data processing system 50 
includes a system unit 52, a video display terminal 54, 
a keyboard 56, and a mouse 58. Data processing sys- 
tem 50 may be implemented utilizing any suitable 
computer such as an IBM PS/2 computer, a product 
of International Business Machines Corporation, lo- 
cated in Armonk, New York. "PS/2" is a registered 
trademark of International Business Machines Cor- 
poration, located in Armonk, New York. Although the 
depicted embodiment involves a personal computer, 
a preferred embodiment of the present invention may 
be implemented in other types of data processing 
systems, such as for example, intelligent work sta- 
tions or mini-computers. 

Those skilled in the art will appreciate that the 
hardware depicted in Figure 1 may vary for specific 
applications. For example, other peripheral devices 
such as: optical disk media, audio adapters, and the 
like also may be utilized in addition to the hardware 
depicted. Data processing system 50 may include vid- 



eo storage devices (not shown) for storing digital data 
sets in either analog or digital formats. Examples of 
such video storage devices are laser disk players and 
video recording devices. Audio storage devices, such 

5 as compact disk players, also may be included in data 
processing system 50. 

Figure 2 is a high level block diagram further il- 
lustrating details of a preferred data processing sys- 
tem 50 as shown in Figure 1 . Data processing system 

10 50 is controlled primarily by software executed within 
central processing unit (CPU) 202. CPU 202 is cou- 
pled to display 204, and receives user input from user 
input device 206. CPU 202 is also coupled to memory 
208 and one or more direct access storage devices 

15 (DASDs) depicted at block 210. Memory 208 and 
DASD 210 may be used for storing data sets com- 
prised of digital representations of text, graphics, 
sound, animated graphics, synthesized speech, and 
video. User in put device 206 may be a mouse, a touch 

20 sensitive tablet or screen, a joy stick, a track ball, or 
a screen activated light pen. CPU 202 also coupled to 
audio output device 21 2 and peripheral controller 21 4. 
Audio output device 212, which may include an am- 
plifier and speaker system, may be utilized to present 

25 an audio portion of a multimedia presentation. Periph- 
eral controller 214 may be utilized to control peripher- 
al devices, such as a video player/recorder, a slide 
projector or a laser disk player, during a multimedia 
presentation. 

30 During a multimedia presentation, CPU 202 may 

access data sets comprising digital representations of 
text, graphics, animated graphics, and video, from 
memory 208 or DASD 210, and utilize display 204 to 
display the visual portion of a multimedia presenta- 

35 tion. CPU 202 may also access data sets comprising 
digitized sound or data used to produce synthesized 
speech, and utilize audio output device 212 to present 
the aural portion of a multimedia presentation. 

Those persons skilled in the art will recognize 

40 that other media, besides aural and visual media, 
may be incorporated into a multimedia presentation. 
For example, the medium of motion may be utilized to 
commu nicate the feel ing of f I ig ht to t he user of a f I ig ht 
simulator. A data set representing commands to move 

45 the flight simulator in a predetermined fashion may be 
accessed by CPU 202, in response to input from user 
input device 206, and then transmitted, via peripheral 
controller 214, to the mechanism which moves the 
flight simulator. 

50 Turning now to Figure 3, there is depicted display 

204 at three sequential points in time during a multi- 
media presentation. Figure 3a illustrates first running 
person 302 entering the field of view on display 204. 
Although depicted in stick figure format within Figure 

55 3, those skilled in the art will appreciate that each per- 
son depicted therein may be presented as part of a 
video presentation. First running person 302 is an ani- 
mated element within the multimedia presentation. 



3 



5 



EP 0 596 823 A2 



6 



Figure 3b depicts two animated elements: first 
running person 302 leaving the field of view, and sec- 
ond running person 304 entering the field of view. 
Graphic indicator 306, which is associated with, and 
following the motion of, second running person 304, 
may be utilized to indicate the presence of an asso- 
ciated data set stored within data processing system 
50 which is also currently relevant to this portion of 
the multimedia presentation. Such an associated 
data set may, in this example, contain additional in- 
formation concerning second running person 304, 
and may be in the form of text, graphics, sounds, ani- 
mated graphics, synthesized speech, or video. A data 
processing system user may choose to view or listen 
to the associated data set if the user requests the ad- 
ditional data set be presented, within a predeter- 
mined period of time during the multimedia presenta- 
tion. The predetermined period of time in which a user 
may requests to view an additional data set may be 
the period of time in which graphic indicator 306 is dis- 
played and associated with an animated element, 
such as second running person 304. 

Graphic button 31 0 may be utilized in conjunction 
with, or as an alternative to, graphic indicator 306 to 
indicate the presence of a relevant associated data 
set. Graphic button 310 may also be utilized in con- 
junction with, or as an alternative to, graphic indicator 
306 as a means to indicate the active portion of dis- 
play 204 wherein a user may elect to have a relevant 
associated data set presented by selecting the active 
portion of display 204 utilizing a pointing device, such 
as a mouse. 

Those persons skilled in the art will also recog- 
nize that sound may be audibly variable or aurally ani- 
mated and therefore, music may be considered an 
animated element within a multimedia presentation. If 
the animated element having an associated data set 
is sound, the graphic indicator utilized to indicate the 
presence of an associated data set may be an appro- 
priate button, such as graphic button 310, graphically 
displayed on display 204. 

Figure 3c depicts a point in the multimedia pre- 
sentation after the point illustrated in Figure 3b, where 
second running person 304 is leaving the field of view 
and no longer has graphic indicator 306 following sec- 
ond running person 304. Also shown entering display 
204 is third running person 308. Graphic indicator 306 
is displayed when a data set associated with an ani- 
mated element, such as second running person 304, 
exists and when the associated data set is relevant to 
what is transpiring in the multimedia presentation. 

Referring now to Figure 4, there is depicted a 
high level logical flow chart illustrating a programmed 
response of the data processing system of Figure 1 
in accdrdance with a preferred embodiment of the 
method and system of the present invention during a 
set-up operation. Terminal block 400 represents the 
entry point to the process. First, as illustrated at block 



402, data sets, which may include a multimedia pre- 
sentation and the data to be associated with an ani- 
mated element within the multimedia presentation, 
are created or identified within data processing sys- 

5 tern 50. Next, as illustrated at block 404, the data sets 
are stored within data processing system 50. Block 
406 illustrates the next step of associating the data 
sets with selected animated elements within a multi- 
media presentation. In this example, the multimedia 

10 presentation may be a video of people running in a 
foot race. The associated data set may be statistics, 
or an interview with, a particular runner in the race. 

Next, as depicted at block 408, a time period is 
selected in which a graphic indicator is displayed in 

15 association with selected animated elements to indi- 
cate the presence of an associated data set stored 
within data processing system 50. Typically, the 
graphic indicator is displayed when the associated 
data set is relevant to the activity of the animated ele- 

20 ment at that particular time in the multimedia presen- 
tation. The graphic indicator may, for example, be a 
rectangle surrounding and following the animated 
element, as shown in Figure 3b by graphic indicator 
306, or the graphic indicator may include a graphic 

25 button placed in display 204. The graphic indicator 
may also be a blinking element. User selection of an 
additional data set may be accomplished by utilizing 
a graphic button in association with a graphic indica- 
tor, such as graphic indicator 306. 

30 In addition to selecting a time to display a graphic 

indicator, as depicted at block 408, a period of time 
may be selected in which a user s attempt to select 
an associated data set is effective, as depicted at 
block 410. The period of time during the multimedia 

35 presentation in which the associated graphic indica- 
tor is displayed with an animated element is not nec- 
essarily the same period of time in which a user may 
select to view or listen to the associated data set. Fi- 
nally, terminal block41 2 represents the end of the set 

40 up process. 

With reference now to Figure 5, there is depicted 
a hih level logical flow chart which illustrates a pro- 
grammed response of the data processing system of 
Figure 1 in accordance with a preferred embodiment 

45 f the method and system of the present invention dur- 
ing a graphic indication process. Terminal block 500 
represents the entry point to the process. First, as de- 
picted at block 502, a multimedia presentation is start- 
ed. A multimedia presentation may communicate data 

50 sets stored within data processing system 50 in sev- 
eral formats, including: text, graphics, sound (includ- 
ing hi-fidelity stereo), animated graphics, synthe- 
sized speech, and video. As the multimedia presen- 
tation continues, the process continuously determi- 

55 nes whether or not an element having an associated 
data set is present within display 204, as illustrated at 
decision block 504. If animated elements having as- 
sociated data sets are not present, the process deter- 
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mines whether or not the multimedia presentation has 
been completed, as depicted at decision block 506. If 
the multimedia presentation has been completed, the 
process ends, as illustrated terminal block 508. If the 
multimedia presentation has not been completed, the 
process continues presenting the multimedia presen- 
tation, as depicted at process block 510. 

Referring again to block 504, and after determin- 
ing that an animated element having an associated 
data set is present within the display, the process 
next determines whether or not the data set associ- 
ated with the animated element is relevant at this par- 
ticular point in the multimedia presentation, as illu- 
strated at decision block 512. Selected animated ele- 
ments will always be associated with particular data 
sets, but such data sets may not always be relevant 
to the present activity portrayed in the multimedia 
presentation. If the associated data set is not relevant 
at this point in the multimedia presentation, the proc- 
ess continues to look for animated elements within 
the display having associated data sets, as illustrated 
at decision block 504. 

If the element having an associated data set is 
present within display 204 and the associated data 
set is currently relevant, the process illustrates graph- 
ically that the animated element has a relevant asso- 
ciated data set, as depicted at block 514. As dis- 
cussed previously, the graphic indication may be a 
rectangle surrounding, and moving with, the selected 
animated element, as shown in Figure 3b by graphic 
indicator 306. Next, the process permits the user to 
effectively select additional data sets associated with 
the selected animated element, as illustrated at proc- 
ess block 516. The process then determines whether 
or not the period for graphically indicating the exis- 
tence of an associated data set has expired, as de- 
picted at decision block 51 8. If such an indication per- 
iod has not expired, then effective user selection is 
permitted as illustrated at block 516. If such an indi- 
cation period has expired, then the process removes 
the graphic indication associated with the selected 
animated element, as illustrated at block 520. The ef- 
fective user selection period may or may not coincide 
with the period of time the graphic indication is asso- 
ciated with the selected animated element. There- 
fore, the process must separately determine whether 
or not the user selection period has expired, as depict- 
ed at decision block 522. If the user selection has not 
expired, effective user selection is permitted, as illu- 
strated at block 524. 

If the user selection period has expired, then the 
process makes further attempts by the user to select 
the associated data set ineffective, as depicted at 
block 526. After user selection has been made inef- 
fective, as illustrated at block 526, the process deter- 
mines whether or not the multimedia presentation has 
been completed, as depicted at decision block 506. If 
the multimedia presentation has not been completed, 



the process continues presenting the multimedia pre- 
sentation, as illustrated at block 510, and the above 
process is repeated. If the multimedia presentation 
has been completed, the process ends, as illustrated 

5 at terminal block 508. 

Finally, with reference to Figure 6, there is depict- 
ed a programmed response of the data processing 
system 50 wherein the user selection of associated 
data sets is explained in further detail. Terminal block 

w 600 illustrates the starting point for the process. The 
process first determines whether or not a user has 
made an effective selection, as depicted at decision 
block 602. An effective user selection can only be 
made during a selected period of time as determined 

15 at process block 410 of Figure 4. If the user attempts 
to make a selection during any other period of time, 
the selection is considered ineffective and the proc- 
ess will continue waiting for an effective selection, as 
shown at decision block 602. 

20 If an effective user selection has been made, the 

multimedia presentation is paused, as illustrated at 
block 604. Then, a list of associated data sets is dis- 
played so the user may make a selection, as depicted 
at block 606. After displaying such a list, the process 

25 waits for the user to select an associated data set, as 
illustrated at decision blocks 608 and 610. If the user 
decides to cancel the selection of an associated data 
set, the process then determines whether the multi- 
media presentation has been completed, as depicted 

30 at decision block 612. If the user has selected an as- 
sociated data set, then the process presents the as- 
sociated data set, as illustrated at block 614. Presen- 
tation of the associated data set may be accomplish- 
ed by opening a separate window on display 204 and 

35 presenting the associated data set which may com- 
prise a variety of multimedia formats. After the asso- 
ciated data set is presented, the process determines 
whether or not the multimedia presentation has been 
completed, as depicted at decision block 612. If the 

40 multimedia presentation has been completed, the 
process ends as illustrated at terminal block 614. If 
the multimedia presentation has not been completed, 
the multimedia presentation continues, as depicted at 
block 6 1 6, and t he process waits for t he next effective 

45 user selection, as illustrated at decision block 602. If 
the user makes another effective selection, the proc- 
ess described above repeats. 

While the invention has been particularly shown 
and described with reference to a preferred em bod i- 

50 ment, it will be understood by those skilled in the art 
that various changes in form and detail may be made 
therein without departing from the spirit and scope of 
the invention. 

55 

Claims 

1. A method in a data processing system for selec- 
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tively associating stored data with an animated 
element depicted in a multimedia presentation 
within said data processing system, said method 
comprising the steps of: 

storing at least one data set within a data 
processing system; 

associating said at least one data set with- 
in said data processing system with a selected 
animated element within a multimedia presenta- 
tion; 

graphically indicating within said multime- 
dia presentation the existence of said at least one 
data set and said association between said at 
least one data set and said selected animated 
element, wherein a data processing system user 
may be informed of the existence of said at least 
one data set; 

retrieving and presenting said at least one 
data set in response to an input from a user; and 

temporarily suspending said multimedia 
presentation in response to said retrieval and pre- 
senting said at least one data set. 

2. A method in a data processing system for selec- 
tively associating stored data with an animated 
element depicted in a multimedia presentation 
according to claim 1 wherein said step of retriev- 
ing and presenting said at least one data set in re- 
sponse to an input from a user is permitted only 
during a selected period of time. 

3. A method in a data processing system for selec- 
tively associating stored data with an animated 
element depicted in a multimedia presentation 
according to claim 1 wherein said step of graphi- 
cally indicating within said multimedia presenta- 
tion the existence of said at least one data set and 
said association between said at least one data 
set and said selected animated element compris- 
es the step of temporarily displaying an additional 
graphical element in association with said select- 
ed animated element. 

4. A method in a data processing system for selec- 
tively associating stored data with an animated 
element depicted in a multimedia presentation 
according to claim 2 wherein said step of graphi- 
cally indicating within said multimedia presenta- 
tion the existence of said at least one data set and 
said association between said at least one data 
set and said selected animated element compris- 
es displaying an additional graphical element in 
association with said selected animated element 
only during said selected period of time. 

5. A method in a data processing system for selec- 
tively associating stored data with an animated 
element depicted in a multimedia presentation 



according to claim 3 wherein said step of tempor- 
arily displaying an additional graphical element in 
association with said selected animated element 
comprises the step of displaying a rectangle 
5 which substantially surrounds said selected ani- 

mated element. 

6. A method in a data processing system for selec- 
tively associating stored data with an animated 

10 element depicted in a multimedia presentation 

according to claim 1 wherein said step of asso- 
ciating said at least one data set within said data 
processing system with said selected animated 
element includes the step of associating a plural- 

15 ity of data sets within said data processing sys- 

tem with said selected animated element, and 
further including the step of retrieving and pre- 
senting a user with a choice of said plurality of as- 
sociated data sets in response to an input from a 

20 user. 

7. A system for selectively associating stored data 
with an animated element depicted in a multime- 
dia presentation within a data processing system 

25 comprising: 

means for storing at least one data set 
within a data processing system; 

means for associating said at least one 
data set within said data processing system with 
30 a selected animated element within a multimedia 

presentation; 

means for graphically indicating within 
said multimedia presentation the existence of 
said at least one data set and said association be- 
35 tween said at least one data set and said selected 

animated element, wherein a data processing 
system user may be informed of the existence of 
said at least one data set; 

means for retrieving and presenting said at 
40 least one data set in response to an input from a 

user; and 

means for temporarily suspending said 
multimedia presentation in response to said re- 
trieval and presenting said at least one data set. 

45 

8. The system for selectively associating stored 
data with an animated element depicted in a mul- 
timedia presentation within a data processing 
system according to claim 7 wherein said means 

50 for retrieving and presenting said at least one 

data set in response to an input from a user per- 
mits retrieving and presenting said at least one 
data set in response to an input from a user only 
during a selected period of time. 

55 

9. The system for selectively associating stored 
data with an animated element depicted in a mul- 
timedia presentation within a data processing 
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system according to claim 7 wherein said means 
for graphically indicating within said multimedia 
presentation the existence of said at least one 
data set and said association between said at 
least one data set and said selected animated 5 
element comprises means for temporarily dis- 
playing an additional graphical element in asso- 
ciation with said selected animated element. 

10. The system for selectively associating stored 10 
data with an animated element depicted in a mul- 
timedia presentation within a data processing 
system according to claim 8 wherein said means 

for graphically indicating within said multimedia 
presentation the existence of said at least one 15 
data set and said association between said at 
least one data set and said selected animated 
element comprises means for displaying an addi- 
tional graphical element in association with said 
selected animated element only during said se- 20 
lected period of time. 

11. The system for selectively associating stored 
data with an animated element depicted in a mul- 
timedia presentation within a data processing 25 
system according to claim 9 wherein said means 

for temporarily displaying an additional graphical 
element in association with said selected animat- 
ed element comprises means for displaying a rec- 
tangle which substantially surrounds said select- 30 
ed animated element. 

12. The system for selectively associating stored 
data with an animated element depicted in a mul- 
timedia presentation within a data processing 35 
system according to claim 7 wherein said means 

for associating said at least one data set within 
said data processing system with a selected ani- 
mated element includes means for associating a 
plurality of data sets within said data processing 40 
system with a selected animated element, and 
further including means for retrieving and pre- 
senting a userwith a choice of said plurality of as- 
sociated data sets in response to an input from a 
user. 45 
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